Systems and methods for improving the ranking of news articles

ABSTRACT

A system ranks results. The system may receive a list of links. The system may identify a source with which each of the links is associated and rank the list of links based at least in part on a quality of the identified sources.

FIELD OF THE INVENTION

The present invention relates generally to communication systems and,more particularly, to systems and methods for improving the ranking ofnews articles in a communications system.

BACKGROUND OF THE INVENTION

Networks, such as the Internet, have become an increasingly importantpart of our everyday lives. Millions of people now access the Interneton a daily basis to shop for goods and services, obtain information ofinterest (e.g., movie listings, news, etc.), and communicate withfriends, family, and co-workers (e.g., via e-mail or instant messaging).

Currently, when a person wishes to purchase a product or simply findinformation on the Internet, the person enters into his/her web browsera Uniform Resource Locator (URL) pertaining to a web site of interest inorder to access that particular web site. The person then determineswhether the information of interest is available at that particular website.

For example, suppose a person wishes to obtain the latest news regardinga particular topic via the Internet. The person accesses a web site thatincludes a conventional search engine. The person enters one or moreterms relating to the topic of interest, such as “Iraq,” into the searchengine to attempt to locate a news source that has published an articlerelating to the topic. Using a search engine in this manner to locateindividual web sites that provide news articles relating to the desiredtopic often results in a ranked list of hundreds or even thousands of“hits,” where each hit may correspond to a web page that relates to thesearch term(s).

While each of the hits in the ranked list may relate to the desiredtopic, the news sources associated with these hits, however, may not beof uniform quality. For example, CNN and BBC are widely regarded as highquality sources of accuracy of reporting, professionalism in writing,etc., while local news sources, such as hometown news sources, may be oflower quality.

Therefore, there exists a need for systems and methods for improving theranking of news articles based on the quality of the news source withwhich the articles are associated.

SUMMARY OF THE INVENTION

Implementations consistent with the principles of the invention adjustthe ranking of news articles in search results based at least in part onthe quality of the news sources with which the articles are associated.

In accordance with one implementation consistent with the principles ofthe invention, a method for ranking results is provided. The methodincludes receiving a list of links, identifying, for each of the links,a source with which the link is associated, and ranking the list oflinks based at least in part on a quality of the identified sources.

In another implementation consistent with the principles of theinvention, a server includes a processor and a memory that is configuredto store quality indicators for a group of sources. The processor mayreceive a list of objects, identify a source with which each of theobjects is associated, and rank at least one object in the list ofobjects based at least in part on the quality indicator associated withthe source with which the one object is associated.

In a further implementation consistent with the principles of theinvention, a method for determining a quality of a news source isprovided. The method may include determining one or more metric valuesfor the news source based at least in part on at least one of a numberof articles produced by the news source during a first time period, anaverage length of an article produced by the news source, an amount ofimportant coverage that the news source produces in a second timeperiod, a breaking news score, an amount of network traffic to the newssource, a human opinion of the news source, circulation statistics ofthe news source, a size of a staff associated with the news source, anumber of bureaus associated with the news source, a number of originalnamed entities in a group of articles associated with the news source, abreadth of coverage by the news source, a number of different countriesfrom which network traffic to the news source originates, and thewriting style used by the news source. The method may further includecalculating a quality value for the news source based at least in parton the determined one or more metric values.

In yet another implementation consistent with the principles of theinvention, a method for providing search results is provided. The methodmay include receiving a list of objects; identifying a source with whicheach of the objects in the group of objects is associated; determining aquality of each of the identified sources; and ranking each object inthe group of objects based at least in part on the determined quality ofthe source with which the object is associated.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate an embodiment of the inventionand, together with the description, explain the invention. In thedrawings,

FIG. 1 is an exemplary diagram of a system in which systems and methodsconsistent with the principles of the invention may be implemented;

FIG. 2 is an exemplary diagram of the server of FIG. 1 in animplementation consistent with the principles of the invention;

FIG. 3 is an exemplary diagram of a database that may be associated withthe server of FIG. 2 in an implementation consistent with the principlesof the invention;

FIG. 4 is a flow chart of an exemplary process for determining a sourcerank for news sources in an implementation consistent with theprinciples of the invention; and

FIG. 5 is a flow chart of an exemplary process for adjusting a rankingof objects in an implementation consistent with the principles of theinvention.

DETAILED DESCRIPTION

The following detailed description of implementations consistent withthe principles of the invention refers to the accompanying drawings. Thesame reference numbers in different drawings may identify the same orsimilar elements. Also, the following detailed description does notlimit the invention.

Implementations consistent with the principles of the invention improvethe ranking of news articles in search results based at least in part onthe quality of the sources associated with the news articles. While theforegoing description focuses on the ranking of news articles from newssources, it will be appreciated that the techniques described herein areequally applicable to improving the ranking of items other than newsarticles.

Exemplary System

FIG. 1 is an exemplary diagram of a system 100 in which systems andmethods consistent with the principles of the invention may beimplemented. System 100 may include multiple clients 110 connected toservers 120 and 130 via a network 140. Network 140 may include a localarea network (LAN), a wide area network (WAN), a telephone network, suchas the Public Switched Telephone Network (PSTN), an intranet, theInternet, a similar or dissimilar network, or a combination of networks.Two clients 110 and three servers 120/130 have been illustrated asconnected to network 140 in FIG. 1 for simplicity. In practice, theremay be more or fewer clients 110 and/or servers 120/130. Also, in someinstances, a client 110 may perform the functions of a server 120/130and a server 120/130 may perform the functions of a client 110.

Clients 110 may include devices, such as wireless telephones, personalcomputers, personal digital assistants (PDAs), lap tops, etc., threadsor processes running on these devices, and/or objects executable bythese devices. Servers 120/130 may include server devices, threads,and/or objects that operate upon, search, or maintain documents in amanner consistent with the principles of the invention. Clients 110 andservers 120/130 may connect to network 140 via wired, wireless, oroptical connections.

In an implementation consistent with the principles of the invention,server 120 may include a search engine 125 usable by clients 110.Servers 130 may store objects (or web documents) accessible by clients110.

Exemplary Server Configuration

FIG. 2 is an exemplary diagram of server 120 in an implementationconsistent with the principles of the invention. Clients 110 and servers130 may be similarly configured. Server 120 may include a bus 210, aprocessor 220, a main memory 230, a read only memory (ROM) 240, astorage device 250, one or more input devices 260, one or more outputdevices 270, and a communication interface 280. Bus 210 may include oneor more conductors that permit communication among the components ofserver 120.

Processor 220 may include any type of conventional processor ormicroprocessor that interprets and executes instructions. Main memory230 may include a random access memory (RAM) or another type of dynamicstorage device that stores information and instructions for execution byprocessor 220. ROM 240 may include a conventional ROM device or anothertype of static storage device that stores static information andinstructions for use by processor 220. Storage device 250 may include amagnetic and/or optical recording medium and its corresponding drive.

Input devices 260 may include one or more conventional mechanisms thatpermit a user to input information to server 120, such as a keyboard, amouse, a pen, one or more biometric mechanisms, such as a voicerecognition device, etc. Output devices 270 may include one or moreconventional mechanisms that output information to the user, including adisplay, a printer, a speaker, etc. Communication interface 280 mayinclude any transceiver-like mechanism that enables server 120 tocommunicate with other devices and/or systems. For example,communication interface 280 may include mechanisms for communicatingwith another device or system via a network, such as network 140.

As will be described in detail below, server 120, consistent with theprinciples of the invention, may provide search results in response to aquery from a client 110. In one implementation, server 120 modifies newsarticle search results based on the quality of the source providing thenews article. Server 120 may perform these operations in response toprocessor 220 executing software instructions contained in acomputer-readable medium, such as memory 230. A computer-readable mediummay be defined as one or more memory devices and/or carrier waves. Thesoftware instructions may be read into memory 230 from anothercomputer-readable medium, such as data storage device 250, or fromanother device via communication interface 280. The softwareinstructions contained in memory 230 may cause processor 220 to performprocesses that will be described later. Alternatively, hardwiredcircuitry may be used in place of or in combination with softwareinstructions to implement processes consistent with the principles ofthe invention. Thus, the invention is not limited to any specificcombination of hardware circuitry and software.

A server 120, consistent with the principles of the invention, may rankor alter a ranking of search results based on information from one ormore associated databases. The databases may be stored at server 120(e.g., in memory 230) or externally from server 120.

FIG. 3 is an exemplary diagram of a database 300 that may be associatedwith server 120 in an implementation consistent with the principles ofthe invention. While only one database is described below, it will beappreciated that server 120 may be associated with one or moreadditional databases (not shown) stored locally at server 120 ordistributed throughout network 140.

As illustrated, database 300 may include a source field 310 and a sourcerank field 320. Database 300 may include additional fields (not shown)that aid in searching and sorting information in database 300 and/orinformation retrieved from network 140.

Source field 310 may identify news sources in system 100. The newssources, labeled 1 through N, where N is a number greater than or equalto 1, may include local news sources, such as local on-line newspapersor local television station web sites, national news sources,international news sources, specialty news sources (e.g., technical,sports, or entertainment magazines or newspapers), and/or any other typeof source of news from a network, such as the Internet. For example, thenews sources may include on-line versions of the Washington Post, CNN,MSNBC, BCC, the New York Post, USA Today, the Pittsburgh Post-Gazette,ESPN, Sports Illustrated, and the like.

Source rank field 320 may store a value that may be used to adjust theranking of articles (or documents) retrieved from the source identifiedin source field 310. In this way, the ranking of an article from ahigher quality news source may be adjusted to be higher than an articleon the same topic from a lower quality news source. As will be describedin detail below, the value of a given news source may be based at leastin part on the credibility, accuracy of reporting, professionalism inwriting, etc. of the news source.

Exemplary Processing

FIG. 4 is a flow chart of an exemplary process for determining a sourcerank for news sources in an implementation consistent with theprinciples of the invention. The process described below may beperformed automatically by, for example, server 120 or manually by ahuman operator. In alternative implementations, parts of the processdescribed below may be performed automatically while other parts may beperformed manually.

Processing may begin by considering a group of metrics for each newssource (act 405). Each metric may measure a specific attribute of thenews source that can serve as a partial indicator of the quality of thenews source. In one implementation consistent with the principles of theinvention, each metric may be computed as a numerical value, where, forexample, a higher computed value may indicate a higher quality newssource.

As will be described in greater detail below, the group of metrics mayinclude the number of articles produced by the news source during agiven time period, an average length of an article from the news source,the importance of coverage from the news source, a breaking news score,usage pattern, human opinion, circulation statistics, the size of thestaff associated with the news source, the number of news bureausassociated with the news source, the number of original named entitiesthe source news produces within a cluster of articles, the breath ofcoverage, international diversity, writing style, and the like. A firstmetric in determining the quality of a news source may include thenumber of articles produced by the news source during a given timeperiod. The time period may be a week, bi-week, month, or the like. Inone implementation consistent with the principles of the invention, thefirst metric may be determined by counting the number of non-duplicatearticles produced by the news source over the time period. In analternate implementation, the first metric may be determined by countingthe number of original sentences produced by the news source.

A second metric may include an average length of an article from thenews source. The average length may be measured, for example, in wordsor sentences. In one implementation consistent with the principles ofthe invention, the second metric may be determined by determining theaverage length of non-duplicate articles produced by the news source.For example, it may be determined that the average length of an articlefrom CNN is 300 words, while the average article length from AmateurNews Network is 150 words. Therefore, the value of the second metric forCNN may be 300 and for Amateur News Network may be 150.

A third metric may include the importance of coverage by the newssource. This metric may be based at least in part on the assumption thatthe “size” of the underlying news story (referred to hereinafter as“story size”) for a given article can be determined. This metric may bea total value that represents the story size scores for allnon-duplicate articles produced by the news source under considerationduring a fixed time period. The time period may be a week, bi-week,month, or the like. As an example, if D is an article, then the storysize of D can be measured as the number of distinct other articles knownto the system (e.g., server 120) that are on the same subject. Forexample, if D is an article about the crash of the Columbia Shuttle andthere were 500 other distinct articles on the subject, then the storysize would be 500. A number of techniques exist for detecting otherarticles on the story. For example, two such techniques are described incopending, commonly assigned, U.S. patent application Ser. No. ______[Docket No. GOOGLE-00300], filed Jun. 30, 2003, and entitled “Methodsand Apparatus for Ranking Documents,” and U.S. patent application Ser.No. ______ [Docket No. GOOGLE-00400], filed Jun. 30, 2003, and entitled“Methods and Apparatus for Clustered Aggregation of News Content,” bothof which are expressly incorporated by reference in their entiretiesherein. A group of related articles is referred to hereinafter as a“cluster.” In one implementation consistent with the principles of theinvention, the metric value may be restricted to the story sizes of thebiggest N stories, where N is a positive integer greater than or equalto 1, covered by the given source over the stipulated time period (e.g.,the biggest 100 stories during 1 week covered by CNN as measured byserver 120).

A fourth metric may include a value representative of a breaking newsscore. This metric may measure the ability of the news source to publisha story soon after an important event has occurred. This metric mayaverage the “breaking score” of each non-duplicate article from the newssource, where, for example, the breaking score is a number that is ahigh value if the article was published soon after the news eventhappened and a low value if the article was published after much timehad elapsed since the news story broke.

In one implementation consistent with the principles of the invention,all articles in the cluster are sorted by publication time in increasingorder and the first article's time is taken as the time of the event.For example, assume T is the difference in time between the currentarticle and the first article. A threshold N1 may be used to denote theinterval after which a story is no longer considered breaking.Therefore, the breaking news score metric may be determined as:

If T>N1, then breaking_score=0;

If 0<T≦N1, then breaking_score=log(N1/T); and

If T=0, then breaking_score=log(N1).

N1 may be represented in hours, such as 3 hours.

In another implementation consistent with the principles of theinvention, all articles in the cluster are sorted by time in increasingorder and the rank of each article is taken as the value T above.Correspondingly, a threshold N2 may be used. Therefore, the breakingnews score metric may be determined as:

If T>N2, then breaking_score=0; and

If 1<T≦N2, then breaking_score=log(N2/T).

In one exemplary implementation, N2 may be 10.

In yet another implementation consistent with the principles of theinvention, the breaking news score determined above may be multiplied bya quantity proportional to the size of the cluster of related articlesto which the given article belongs. For example, the breaking news scoremay be multiplied by a factor=(1+log(cluster size)). This emphasizes thevalue of breaking news when the story is important and is seen to form abig cluster.

In still another implementation, instead of averaging the breaking scorevalues, the values are summed if the cluster size is larger than apredetermined value (e.g., 30). Therefore, in this situation, thebreaking news score may be determined as follows:

-   -   For each article A:        -   If (Size(Cluster(A))>30):

breaking_news[Source(A)]+=30−rank_within_cluster(A).

A fifth metric may include a value representing a usage pattern. Linksgoing from the news search engine's web page to individual articles maybe monitored for usage (e.g., clicks). News sources that are selectedoften are detected and a value proportional to observed usage isassigned. Well known sites, such as CNN, tend to be preferred to lesspopular sites, such as Unknown Town News, which users may avoid. Thetraffic measured may be normalized by the number of opportunitiesreaders had of visiting the link to avoid biasing the measure due to theranking preferences of the news search engine.

A sixth metric may include a value representing human opinion of thenews source. In one implementation consistent with the principles of theinvention, users in general may be polled to identify the newspapers (ormagazines) that the users enjoy reading (or have visited). Alternativelyor in addition, users of the news search engine may be polled todetermine the news web sites that the users enjoy visiting. Theevaluation of news sites by other agencies may be also used (e.g.,newspapers can be compared based at least in part on the number ofPulitzer prizes the newspapers have won, etc.). Moreover, the age of thenews source may be taken as a measure of confidence by the public andmay be used as a metric. In another implementation, evaluators may beshown a selection of articles from individual news sources and asked toassign each source a score. This assigned score may be taken as ametric.

A seventh metric may include a value representing circulation statisticsof the news source. Agencies, such as Media Metrix and NielsenNetratings, publish usage statistics for on-line sites (e.g., newssites). These published traffic numbers can be used as a measure of thequality of a news source. In one implementation consistent with theprinciples of the invention, the circulation statistics of a printnewspaper associated with the news site may be taken as a metric.

An eighth metric may include a value representing the size of the staffassociated with the news source. In one implementation consistent withthe principles of the invention, the staff size may be determined basedat least in part on the number of distinct journalist names recited inarticles from the news source.

A ninth metric may include a value representing the number of newsbureaus associated with the news source.

A tenth metric may include a value representing the number of originalnamed entities the news source produces within a cluster of relatedarticles, averaged, for example, over all articles that have at least Nrelated articles (e.g., N=3). A named entity may correspond to a person,place, or organization. If a news source generates a news story thatcontains a named entity that other articles within the same cluster(hence on the same topic) do not contain, this may be an indication thatthe news source is capable of original reporting. In this analysis, theaverage value added by the given news source in a cluster of a thresholdsize N is evaluated. In one implementation consistent with theprinciples of the invention, the named entities may be considered if noearlier article in the cluster has the same named entity. Named entitiesmay be compared using approximate string matching to compensate forvariations in spelling and abbreviation. Named entities that aremarkedly different from those in other articles may be considered asoriginal.

An eleventh metric may include a value representing breadth (e.g., thenumber of topics on which the source produces content) of the newssource. In one implementation consistent with the principles of theinvention, articles from the news source may be categorized into a setof topics (e.g., arts, music, sports, business, etc.) and the range oftopics may be used as a measure of breadth. Any conventionalclassification technique can be used for categorizing articles intotopics. For example, a classification system from the Machine learningliterature can be used to classify news articles into a chosen set oftopics. In another implementation, the number of sections published bythe news source may be taken as a measure of breadth.

A twelfth metric may include a value representing internationaldiversity of the news source. This metric may measure the number ofcountries from which the news site receives network traffic. In oneimplementation consistent with the principles of the invention, thismetric may be measured by considering the countries from which knownvisitors to the news web site are coming (e.g., based at least in parton the Internet Protocol (IP) addresses of those users that click on thelinks from the search site to articles by the news source beingmeasured). The corresponding IP addresses may be mapped to theoriginating countries based on a table of known IP block to countrymappings. In another implementation, the IP addresses of web sites whichlink to the given news web site may be monitored and the number ofdistinct countries from which the news site is linked may be used as themetric.

A thirteenth metric may include a value representing the writing styleused by the news source. Automated tests for measuring spellingcorrectness, grammar, and reading levels can be used to generate ametric value that reflects writing style. A score proportional to themeasured quality of writing style may then be assigned.

It will be appreciated that other metrics may be considered in additionto or as an alternative to the above group of metrics. For example,another metric may include a value representing the number of hyperlinksto the news web site.

Once a group of metrics has been considered, a source rank for each ofthe news sources may be determined based at least in part on the groupof metrics (act 410). To determine the source rank for a news source,some or all of the group of metrics described above may be combined toproduce a final score (i.e., source rank) for the news source. A numberof techniques may be used to determine the source rank of the newssources. For example, in one implementation consistent with theprinciples of the invention, each metric may be multiplied by acorresponding factor and the resulting values may be totaled to give thesource rank for the news source. Alternatively, each metric may benormalized to be in the range 0 to 1 and the resulting values may betotaled to give a final metric value (i.e., source rank). For example,the normalization may be accomplished by dividing each metric value bythe maximum possible value assigned by that metric.

In another implementation, the average rank of the news source invarious metrics may be computed. For example, if CNN has a rank of 1 incirculation statistics, a rank of 2 in international popularity, and arank of 9 in the number of international bureaus, then considering onlythese metrics CNN has an average rank of

(1+2+9)/3=4.

In still a further implementation, for each news source, the percentilescore on each metric relative to the best news source for that metricmay be used for determining the source rank for the news source. Forexample, if CNN has a rank of 2 for international popularity and BCC hasthe highest rank for that metric of 10, then CNN's percentile score forthat metric may be 0.2.

As an alternative to the above, one of the techniques described abovemay be used with only the best N metrics for the given news source beingconsidered. N may be a positive integer greater than or equal to one. Inone implementation, N may be 5. By considering only the best N metricsfor a given news source, this allows for incorporation of news sourcesfor which some of the metrics have yet to be computed.

Once a source rank has been determined for a news source, the sourcerank may be stored by server 120 (act 415). In one implementation,server 120 may store the identity of the news source and correspondingsource rank value in a database, such as database 300.

FIG. 5 is a flow chart of an exemplary process for adjusting a rankingof objects (e.g., news articles) in an implementation consistent withthe principles of the invention. While the following description focuseson ranking objects retrieved as a result of a search query,implementations consistent with the principles of the invention are notso limited. In fact, implementations consistent with the principles ofthe invention are equally applicable to any system or method that ranksnews articles according to a scoring criterion. The criterion caninclude, for example, a query (e.g., a search engine query, as describedin the exemplary scenario below), a topic (e.g., sports), a list ofkeywords (e.g., keywords from an initial set of search resultdocuments), a geographical area (e.g., New York), a list of articles ina cluster of articles, or an exemplary set of documents.

Processing may begin with a user accessing server 120 (FIG. 1) using,for example, web browser software on a client, such as client 110. Theuser may then provide a query that includes one or more search terms tosearch engine 125 maintained by server 120 (act 505). In oneimplementation, the search query includes one or more terms relating toa news topic. For example, if a user desired to view news articles aboutGeorge Bush, the user may cause client 110 to send a search query toserver 120 with the search terms “George Bush.”

In response to receiving the search query, server 120 may generate aranked list of results in a conventional manner (act 510). The resultsmay include references (e.g., links) to news articles and possibly atextual description of the links. Server 120 may determine, for eachlink in the ranked list, whether the link corresponds to a news sourcefor which a source rank has been determined. To determine whether asource rank exists for a link, server 120 may first identify the newssource to which the link corresponds (act 515). In one implementation,server 120 may identify the news source based at least in part on theuniform resource locator (URL) associated with the link. For example,server 120 may determine that link “www.cnn.com/2003/abc/index.html,”corresponds to the news source “CNN.” Other techniques for identifyingthe news source to which a link corresponds may alternatively be used.

Once the news source has been identified, server 120 may determinewhether a source rank exists for a link by, for example, accessingdatabase 300 and determining whether the news source corresponding tothe link is stored in source field 310 (act 520). If the news sourcedoes not exist in source field 310, server 120 may not adjust theranking of the link. If, on the other hand, the news source exists insource field 310, server 120 may retrieve the source rank of the newssource from source rank field 320. Server 120 may then adjust theranking of the link corresponding to the news source based at least inpart on the retrieved source rank (act 520).

Given an initial ranking R1 of links, server 120 may produce an adjustedranking R2 by computing an improved score for each link. Server 120 maydetermine the new score by combining the score corresponding to theranking in R1 with the source rank for the news source associated withthe link. In one implementation consistent with the principles of theinvention, server 120 may determine the new score as a weighted sum. Forexample, server 120 may determine the new score for a link as follows:

NEWSCORE(D)=alpha*OLDSCORE(D)+beta*SOURCERANK(SOURCE(D))

where SOURCE(D) is the news source of link D and alpha and beta aresuitable constants. For example, in one implementation consistent withthe principles of the invention, alpha may be set to 0.8 and beta may beset to 0.2. It will be appreciated that other values for alpha and betamay alternatively be used. Other techniques for adjusting the ranking ofa link may alternatively be used, such as taking the average of the R1score and the source rank. In this way, an improved ranking of newslinks may be produced.

Once the ranked list has been adjusted, server 120 may provide theadjusted ranked list of links to client 110 (act 525). Server 120 maytransmit the adjusted list of links to client 110 via network 140.

In other implementations consistent with the principles of theinvention, server 120 may, in act 510, retrieve a non-ranked list ofresults in response to receiving the search query. In such instances,server 120 may rank the list of results based on the source rankassociated with the new sources with which the list of results isassociated.

CONCLUSION

Implementations consistent with the principles of the invention mayimprove the ranking of news articles based at least in part on thequality of the news sources associated with the news articles.

The foregoing description of exemplary embodiments of the presentinvention provides illustration and description, but is not intended tobe exhaustive or to limit the invention to the precise form disclosed.Modifications and variations are possible in light of the aboveteachings or may be acquired from practice of the invention. Forexample, the above-described functions need not be performed by server120. In other implementations, one or more of the acts described in FIG.5 may be performed by a client 110. For example, a browser assistant(i.e., software that operates in conjunction with a conventional webbrowser) may perform one or more of the acts described with respect tothe process of FIG. 5.

Moreover, as set forth above, implementations consistent with theprinciples of the invention are not limited to ranking news articles.For example, implementations consistent with the principles of theinvention may be used for ranking other types of items that may beretrieved over a network or from one or more databases.

While series of acts have been described with regard to FIGS. 4 and 5,the order of the acts may be varied in other implementations consistentwith the present invention. Moreover, non-dependent acts may beimplemented in parallel.

No element, act, or instruction used in the description of the presentapplication should be construed as critical or essential to theinvention unless explicitly described as such. Also, as used herein, thearticle “a” is intended to include one or more items. Where only oneitem is intended, the term “one” or similar language is used.

1-31. (canceled)
 32. A computer-implemented method comprising:obtaining, in response to receiving a search query, articles andrespective scores; identifying, using one or more processors, for anarticle of the articles, a source with which the article is associated;determining, using one or more processors, a score for the source, basedon polling one or more users to request the one or more users to providea metric that represents an evaluation of a source; and adjusting, usingone or more processors, the score of the article based on the score forthe source.
 33. The method of claim 32, where the score for the sourceis further based on a breaking score indicating a length of time betweenan occurrence of an event and publication, by the source, of an articleassociated with the event.
 34. The method of claim 32, where identifyingthe source includes identifying the source based on an addressassociated with the article.
 35. The method of claim 32, where adjustingthe score of the article includes: determining, using the score for thesource, a new score for the article associated with the source; andadjusting the score of the article based on the determined new score.36. The method of claim 32, where the score for the source is furtherbased on a usage pattern indicating traffic associated with the source.37. The method of claim 32, where the score for the source is furtherbased on a number of entities, associated with the source, that areassociated with only one article in a cluster of related articles. 38.The method of claim 32, where the score for the source is further basedon a value representing international diversity of the source.
 39. Asystem comprising: one or more server devices to: receive a searchquery; obtain, for the search query, articles and respective scores;identify, using one or more processors, for an article, a source withwhich the score is associated; determine, using one or more processors,a score for the source, the score being based on polling one or moreusers to request the one or more users to provide a metric thatrepresents an evaluation of a source; and adjust, using one or moreprocessors, the score of the article based on the score for the source.40. The system of claim 39, where the score for the source is furtherbased on a breaking score indicating a length of time between anoccurrence of an event and publication, by the source, of an articleassociated with the event.
 41. The system of claim 39, where, whenidentifying the source, the at least one of the one or more serverdevices is further to identify the source based on an address associatedwith the article.
 42. The system of claim 39, where, when adjusting thescore of the article, at least one of the one or more server devices isfurther to: determine, using the determined score for the source, a newscore for the article associated with the source; and adjust the scoreof the article based on the determined new score.
 43. The system ofclaim 39, where the score for the source is further based on a usagepattern indicating traffic associated with the source.
 44. The system ofclaim 39, where the score for the source is further based on a number ofentities, associated with the source, that are associated with only onearticle in a cluster of related articles.
 45. The system of claim 39,where the score for the source is further based on a value representinginternational diversity of the source.
 46. A computer-implemented methodcomprising: obtaining a list of references to articles; identifying,using one or more processors, for an article in the list of articles, asource with which the article is associated; determining, using one ormore processors, a score for the source, the score being based, at leastin part, on a value representing human opinion of the source, the humanopinion being based on a poll of users that identifies user opinions ofthe sources; and ranking, using one or more processors, the article inthe list of articles based on the score for the source.
 47. The methodof claim 46, where the score is further based on a breaking scoreindicating a length of time between an occurrence of an event andpublication, by the source, of an article associated with the event. 48.The method of claim 46, where the score is further based on a breakingscore indicating a length of time between an occurrence of an event andpublication, by the source, of an article associated with the event. 49.The method of claim 46, where the score is further based on a usagepattern indicating traffic associated with the source.
 50. The method ofclaim 46, where the score is further based on a number of entities,associated with the source, that are associated with only one article ina cluster of related articles.
 51. The method of claim 46, where thescore is further based on a value representing international diversityof the source.